Abstract of " Load Management Techniques for Distributed Stream Processing " Load Management Techniques for Distributed Stream Processing Date U˘ Gur C ¸ Etintemel, Reader

of “Load Management Techniques for Distributed Stream Processing” by Ying Xing, Ph.D., Brown University, May 2006. Distributed and parallel computing environments are becoming inexpensive and commonplace. The availability of large numbers of CPU’s makes it possible to process more data at higher speeds. Stream-processing systems are becoming more important, as broad classes of applications require results in real-time. Since load can vary in unpredictable ways, exploiting the abundant processor cycles requires effective load management techniques. Although load distribution has been extensively studied for the traditional pull-based systems, it has not yet been fully studied in the context of push-based continuous query processing. Push-based data streams commonly exhibit significant variations in their rates and burstiness over all time-scales. Even though medium-to-long term load variations can be dealt with using dynamic load migration techniques rather effectively, short-term variations are much more problematic since capturing such variations and reactively migrating load can be prohibitively expensive or even impractical. We need robust solutions whose performances do not degrade significantly in the presence of time-varying load. In this dissertation, we present both a static operator distribution technique and a dynamic operator re-distribution technique for a cluster of stream processing engines. First, for operators that are impractical to be moved between processing engines on the fly, we propose a resilient static operator distribution algorithm that aims at avoiding overload by maximizing the feasible space of the operator distribution plan. Second, for operators with relatively small load migration overheads, we propose a correlation-based dynamic operator distribution algorithm that aims at minimizing end-to-end latency by minimizing load variance and maximizing load correlation. Our experiment results quantify the effectiveness of the proposed approaches and demonstrate that they significantly outperform traditional operator distribution approaches. Load Management Techniques for Distributed Stream Processing by Ying Xing B. Eng., Automation Engineering, Tsinghua University, 1997 Sc. M., Computer Science, Tsinghua University, 1999 Sc. M., Computer Science, Brown University, 2001 Sc. M., Applied Mathematics, Brown University, 2004 A dissertation submitted in partial fulfillment of the requirements for the Degree of Doctor of Philosophy in the Department of Computer Science at Brown University Providence, Rhode Island May 2006 c © Copyright 2006 by Ying Xing This dissertation by Ying Xing is accepted in its present form by the Department of Computer Science as satisfying the dissertation requirement for the degree of Doctor of Philosophy. Date Stan Zdonik, Director Recommended to the Graduate Council Date Uğur Çetintemel, Reader Date John Jannotti, Reader Approved by the Graduate Council Date Sheila Bonde Dean of the Graduate School